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ABSTRACT 


A  graphic  work  station  has  been  created  where  three  aspects 
of  man-machine  interaction  may  be  tested:  (1)  interaction 
with  a  flat  display;  (2)  co-planar  input  and  output;  and 
(3)>  the  ability  to  "see  through  your  hand"  to  a  display 
below.  All  three  are  created  by  a  suitable  arrangement 
of  mirrors  to  direct  a  projected  television  image  to  a 
back-projection  screen  which  is  in  turn  optically  super¬ 
imposed  onto  a  data  tablet  surface  via  a  semi-transparent 
mirror.  Uses  and  benefits  of  the  device  are  demonstrated 
by  means  of  a  set  of  games  which  simulate  a  situation 
where  a  user  must  interact  with  dynamically  changing 
data  and  imagery,  and  annotation  programs  where  rapid  data 
entry  from  hard  copy  is  required. 


1.0  INTRODUCTION 


In  spite  of  the  rapid  advances  in  micro¬ 
electronics  and  computational  power, 
little  has  been  done  to  enhance  the 
human  interface  to  these  systems. 
Available  graphic  input/output  procedures 
still  lack  the  immediacy  that  is  so 
conveniently  provided  by  simple  pencil 
and  paper.  Often,  users  engaged  in  a 
graphic  interaction  must  produce  input 
on  one  device,  such  as  a  digitizing 
tablet,  and  then  see  the  results  of 
that  input  on  a  screen  which  is  neither 
close,  nor  in  the  same  plane,  nor 
capable  of  showing  an  image  at  the 
original  scale.  Additionally,  when 
the  viewing  screen  is  small,  it  may 
be  flat;  when  it  is  either  larger  or 
in  color,  the  display  is  necessarily 
curved,  adding  to  the  dissimilarity 
of  the  input  and  the  presentation.  In 
response  to  such  input/display  problems 
we  developed  a  device  that  duplicates 
some  of  the  salient  features  of  hard 
copy,  non-computer  oriented  graphic 
devices— flatness,  conjoint  input/output 
nature,  and  color  capability.  One 
salient  feature  of  the  device  is  that  it 
provides  the  ability  for  a  user  to 
"see  through  his  or  her  hand"  to  a 
display  underneath. 


Originally  inspired  by  a  work  station 
developed  by  Knowlton  (1)  to  provide 
a  programmable  telephone  operator's 
console,  the  device  consists  of  a  flat 
television  display  optically  superimposed 
upon  either  a  standard  digitizing  tablet 
or  a  high  resolution  touch  panel.  A 
semi-silvered  mirror  interposed  between 
tablet  and  display  provides  parallax-free 
coincidence  of  the  work  surface  with  the 
display  screen  so  that  a  hand  or  object 
on  the  surface  does  not  intersect  the 
television  display ' s  light  path;  thus, 
a  transparency  effect  is  possible.  The 
device  is  illustrated  in  figure  1  and 
schematized  in  figure  two.  Its  operation 
will  be  discussed  in  detail  later. 

Three  separate  and  distinct  benefits 
accrue  from  this  arrangement  of  input 
and  output  surfaces.  The  first,  and 
primary  one,  is  the  absolute  lack  of 
parallax  between  display  screen  and 
data  tablet.  Unlike  stations  which  use 
a  transparent  tablet  overlaid  on  a 
television  monitor  with  an  image  up  to 
1/2  inch  behind  the  tablet,  this  device 
permits  the  image  to  appear  above, 
below,  or  precisely  on  the  working  surface 
When  a  user  picks  a  point  with  either 
his  finger  or  a  stylus,  he  touches  that 
point.  Accurate  feedback  is  provided  by 
direct  presence  and  visual  coincidence, 
rather  than  by  a  cursor.  Additionally, 


the  size  of  the  display  is  the  same  as 
the  active  tablet  area,  therefore  a  motion 
on  one  corresponds  exactly  to  a  motion 
on  the  other. 

The  second  advantage  is  the  creation  of 
a  flat,  full  color  work  station.  Since 
all  available  direct  view  color  monitors 
have  either  spherically  or  cylindrically 
curved  front  surfaces,  users  of  color 
displays  have  accepted  this  curvature 
as  an  unnoticed  and  necessary  inconvenience 
When  the  screen  is  small,  the  resulting 
distortion  is  not  great,  but  as  the  size 
grows,  the  image  appears  more  distorted, 
often  requiring  a  user  to  move  his 
head  to  accurately  view  detail  in  the 
corners . 

The  third  and  least  explored  benefit 
which  should  be  emphasized  is  that 
the  user's  hand  does  not  occlude  the 
display.  In  normally  encountered 
graphical  stations,  the  impulse  to  remove 
one's  hand  and  inspect  the  result  of 
an  action  is  ingrained.  With  this  device, 
that  motion  is  no  longer  necessary.  A 
degree  of  interaction  with  the  system 
never  possible  before  is  now  available. 

One  may  rest  hands,  arms  and  drawing 
aids  on  the  work  surface  without 
interfering  with  the  visibility  of  the 
work.  In  fact,  the  relative  visibility 
of  any  of  these  things  to  the  television 


image  is  readily  controllable  by 
adjustment  of  the  hand  illumination 
lights:  when  they  are  bright,  the  hands 
and  overlays  or  objects  on  the  working 
stfrface  are  most  clearly  seen;  when 
they  are  off,  they  totally  disappear. 

Also,  it  is  possible  to  visually 

mix  computer  output  with  hard  copy  input. 

In  fact,  this  may  be  the  best  application 
of  the  device  as  will  be  explained  later. 

Although  separate,  all  three  features 
of  the  station  work  together  to  support 
each  other.  The  feasibility  of  seeing 
through  the  hand  reinforces  the  lack  of 
parallax  created  by  the  coplanar  input/ 
output  surfaces.  One  no  longer  must 
"look  around"  the  hand,  or  pen.  Likewise, 
the  flat,  similarly  scaled  image  supplements 
the  cursorless  tablet  and  permits  use 
of  touch  panels  at  full  system  resolution. 

2.0  TECHNICAL  DETAILS 

With  the  exception  of  the  television 
projector,  (GE  Light  Valve)*,  the  entire 
apparatus  of  the  work  station  is  contained 
in  a  single  structure.  The  elements  of 
it  are  (1)  a  back  projection  screen  on 
which  the  output  image  is  displayed; 

1  Trade  names  are  used  only  for  precision 
in  reporting  and  do  not  constitute  endorsement 
by  the  Army  or  the  U.S.  Army  Research  Institute 


(2)  a  digitizing  tablet,  which  is  the 
work  surface;  (3)  a  super imposition 
mirror,  which  optically  overlays  the 
display  image  onto  the  tablet  surface; 

(4)  tablet  illumination  lights;  (5) 
suitable  directing  mirrors  to  allow 
the  projection  display  to  be  focused 
onto  the  back  projection  screen  at 
the  appropriate  angle  and  size. 

In  normal  use,  one  sits  in  front  of  the 
station  and  views  both  the  display  and 
the  tablet  from  a  point  above  the  mirror. 
The  tablet  is  used  by  placing  one's 
hands  below  the  mirror  and  working 
directly  on  the  surface.  The  tablet  is 
thus  seen  through  the  mirror,  and  the 
screen  by  reflection  from  it. 

The  orientation  of  the  mirror  relative 
to  the  two  surfaces  determines  the 
apparent  angular  orientation  of  the  display 
and  tablet  images.  When  the  mirror 
exactly  bisects  the  angle  formed  by 
these  surfaces,  they  appear  to  lie  in 
parallel  planes — at  the  same  angle 
and  orientation. 

The  vertical  placement  of  the  mirror 
determines  the  apparent  height  of  the 
display  screen.  When  the  mirror  is 
midway  between  the  screen  and  the  tablet 
surface  (i.e.,  when  its  plane  intersects 
the  vertex  of  the  angle  formed  by  the 
surfaces) ,  the  display  image  and  tablet 

-  5  - 


\ 

SCMENN 


surface  appear  in  precisely  the  same 
plane,  in  perfect  register.  This 
registration  exists  for  all  viewing 
positions  above  the  mirror. 

The  relative  visibility  of  either 
surface  is  controlled  by  their 
relative  brightness  and  the  transparency 
of  the  mirror.  Projector  brightness 
is  controlled  by  direct  adjustment  of 
the  projector.  Tablet  brightness  is 
controlled  by  adjustment  of  a  set  of 
lights  placed  below  the  mirror  and  aimed 
downward,  at  the  tablet  surface.  Normally 
the  tablet  itself  is  painted  matte 
black  so  that  no  background  illumination 
dilutes  the  display.  Objects  on  it, 
such  as  pens  and  hands,  reflect  light, 
and  show  up  clearly  through  the  mirror. 

2.1  The  Back  Projection  Screen 

In  Knowlton's  original  implementation 
of  the  telephone  operator's  console, 
a  standard  television  monitor  provided 
the  display  image.  This  had  the 
disadvantage  that  the  resulting  image 
was  curved,  necessitating  a  similarly 
curved  keyboard  below.  In  this  work, 
we  required  a  flat  display  surface, 
so  that  the  image  would  conform  to  an 
already  flat  tablet,  and  because  the 
flatness  was  perceived  as  mandatory  for 
satisfactory  direct  interaction  with  graphics. 


A  curved  screen  and  a  curved  tablet,  even 
if  available,  would  have  resulted  in 
an  extremely  unfamiliar  working  surface, 
and  one  not  readily  amenable  to  digitization 
of  hard  copy.  We  thus  chose  to  use  a 
rear  projected  image  and  experimented 
with  a  wide  variety  of  screens  and 
correction  lenses  to  generate  uniform 
image  brightness  across  the  entire 
display. 


Design  of  a  rear  projection  system  is 
optimized  for  a  given  resolution, 
viewing  area,  and  brightness,  and  the 
parameters  that  are  controllable  are 
the  gain  of  the  screen  and  its  inherent 
resolution.  Gain  is  a  measure  of 


the  apparent  brightness  as  a  function 
of  the  angle  of  view.  A  low  gain 
screen  disperses  lights  more  evenly 
over  wider  angles,  and  thus  permits 
the  largest  viewing  area.  The  entire 
surface  appears  uniformly  bright. 

Better  resolution  is  obtained  with 
high  gain  screens.  However,  high  gain 
screens  provide  proportionately  more 
light  in  the  direction  of  projection, 
causing  image  brightness  to  fall  off  at 
the  edges.  In  this  system,  uniform 
brightness  is  especially  important, 
since  it  is  the  relative  screen/tablet 
brightness  that  determines  the  degree  . 
of  transparency  of  the  hand  and  hard  copy. 


I 
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If  the  screen  appears  dimmer  at  the  edges, 
the  brighter  appearance  of  the  hand 
washes  out  the  display  image  and  seriously 
interferes  with  the  utility  of  the  station. 
Since  high  resolution  was  desirable  in 
addition  to  brightness  requirements,  there 
was  a  problem. 

The  solution  adopted  was  a  high  gain 
screen  in  combination  with  a  fresnel 
corrector  plate.  The  correction  lens 
redirects  the  divergent  projection  beam 
to  a  distant  focus,  and  results  in  an 
evenly  bright  screen  over  a  somewhat 
reduced  viewing  area.  In  arriving  at 
this  solution,  a  variety  of  screens 
and  lenses  were  tested.  Ultimately  a 
gain  of  2.5,  in  conjunction  with 
a  fresnel  with  a  focal  length  of  50 
inches  proved  optimal. 

2 . 2  The  Transparency  Mirror 

The  superimposition  mirror  is  critical, 
since  it  creates  the  basic  effect 
unique  to  this  system  and  it  determines 
the  relative  brightness  range  of  the 
teds  let  vs.  the  screen.  Glass  mirrors 
were  obtained  with  various  silver  coatings 
that  resulted  in  transparency /reflectivity 
coefficients  of  75/15;  60/35;  45/45;  these 
were  used  to  explore  the  range  of  effects. 
In  general,  a  sheet  of  glass  can  be 
coated  for  any  degree  of  reflectivity 
desired.  The  screen  selected  had  a 
half  silvered  mirror. 
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The  mirror  is  placed  so  that  the 
silvered  side  is  u£.  Thus,  the 
display  screen  is  seen  through  first 
surface  reflection  and  the  tablet  is 
seen  through  the  glass.  The  reverse 
arrangement  is  impractical  since  the 
mirror  often  is  viewed  at  a  small 
angle  of  incidence  where  any  internal 
glass  reflections  would  lower  the 
effective  resolution  of  the  screen. 

The  height  of  the  mirror  is  adjustable 
so  that  a  wide  variety  of  tablet  surfaces 
and  thicknesses  can  be  used  without 
sacrificing  superimposition  of  the 
surfaces. 

2. 3  The  Tablet 


Two  digitizing  tablets  were  used.  One 

was  a  standard  twenty  inch  square  data 

tablet  (Summagraphics) ;  the  other  a 

touch  panel  specially  designed  for  the 

* 

project  by  Elographics.  Both  were 
painted  black  so  that  their  actual 
surface  was  invisible,  and  in  the  case 
of  the  Summagraphics,  the  pen  body  and 
its  connecting  wire  were  similarly 
painted.  The  appearance  of  a  virtual 
writing  surface  is  created  by  the  display 
image,  not  the  actual  brightness  of  the 
tablet  surface,  so  it  need  not  reflect 
any  light  of  its  own. 

^Biographies  Corp. ,  Oak  Ridge,  Tennessee 
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Of  the  two  devices,  the  touch  panel 
provides  the  more  interesting  application. 
In  normal  use,  a  touch  panel's  full 
resolution  is  rarely  used  either  because 
there  is  no  way  to  align  a  touch  with 
a  display  datum  due  to  parallax,  or 
because  a  tracking  cursor  is  not  possible. 
Additionally,  the  resolution  capability 
of  a  finger  is  often  underestimated 
because  of  its  size  and  blunt  end.  We 
felt  that  the  ability  to  see  through  the 
finger  would  allow  precise  positioning. 


3.0  ILLUSTRATIVE  USES 

We  tested  the  device  (informally  so 
far)  by  two  methods.  In  the  first,  a 
series  of  "games"  was  played  in  which 
the  features  of  coincidence  and  transparency 
were  evaluated;  in  the  second,  the  ability 
of  users  to  make  annotations  superimposed 
on  hard  copy  was  assessed. 

3.1  Games 


Three  games  were  developed  for  this  test. 
Their  common  aim  was  to  compare  a  user's 
ability  to  react  to  dynamically  changing 
display  data.  The  common  underlying 
feature  in  all  these  tests,  besides  their 
ability  to  measure  the  difference  between 
reactions  to  a  separate  tablet/display 
versus  a  coincident  one,  is  their  relation 
to  dynamic  command  and  control  situations. 
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They  are  played  as  follows: 

HIT:  In  this  game,  targets  appear, 
randomly  placed  and  at  random  intervals. 

The  player  must  touch  each  as  it  appears 
and  is  scored  on  the  number  accurately 
touched  within  a  set  time  interval.  This 
game  is  a  paradigm  for  an  active  command 
and  control  station  where  rapid  reaction 
to  dynamically  changing  situations  is 
necessary.  Possible  applications 
derived  from  this  game  include  air 
traffic  control  operations  and  military 
situation  analysis.  The  fact  that  a 
user  can  see  an  object  under  the  hand  and 
that  the  hand  can  move  directly  to  a 
point  on  the  display  without  the  aid  of 
cursors  is  considered  important. 

TRACK:  In  this  game,  one  of  several 
moving  targets  must  be  followed  either 
with  the  tablet  pen  or  the  finger 
(depending  on  which  input  device  is  in 
use) .  The  relative  benefits  of  having 
the  display  and  input  surface  coincident 
can  be  assessed,  since  an  object  moving 
in  a  given  direction  requires  a  user 
motion  in  the  same  direction.  Likewise, 
the  ability  to  see  through  one's  hand 
allows  motion  in  directions  that  are  not 
simple  or  intuitively  clear  with  alternative 
displays. 


4 . 0  EVALUATIONS 


Although  only  informal  testing  was 
conducted,  several  intuitive  guidelines 
and  implications  of  the  work  have 
emerged. 

Annotations  appear  to  be  a  particularly 
useful  application  of  the  device,  the 
only  difficulty  being  the  creation  of 
negative  hard  copy  images.  Of  primary 
advantage  is  the  co-planar  nature  of 
the  input  surface  and  the  display  image. 
People  unskilled  at  drawing  had  little 
difficulty  tracing  the  map,  and  appreciated 
the  directly  visible  feedback.  This 
is  contrasted  with  separate  tablet/ 
display  combinations  where  continually 
"looking  up"  at  the  display  screen  to 
check  progress  and  verify  input  is 
disorienting  and  confusing.  The  parallax- 
free  nature  of  the  device  and  the  similarity 
of  scale  also  help  simplify  the  task. 

The  dynamic  tests  also  were  successful. 

Lab  personnel  consistently  were  able  to 
touch  appearing  targets  faster  with  the 
seeing  through  your  hand  system  than 
with  a  separate  tablet  and  display. 

In  this  case,  the  scale  is  the  primary 
advantage — it  is  simple  to  move 
directly  to  the  place  where  an  object 
appears,  and  difficult  to  make  a 
proportional  motion  on  a  disjoint  surface. 


This  success  is  mitigated  however,  by 
the  similar  success  obtained  with 
transparent  tablets  overlaid  on  a 
curved  screen.  These  tests  do  not 
require  the  high  resolution  capabilities 
of  the  system,  but  merely  prove  the 
value  of  having  the  display  at  least 
close  to  the  tablet,  and  nearly  co-planar. 
Further,  the  device  does  not  greatly 
enhance  the  user's  ability  to  track 
an  object  once  that  object  is  initially 
touched. 

In  general,  users  readily  acclimated 
themselves  to  the  unfamiliar  setting 
and  became  comfortable  with  the  system. 
Gradually,  users  learned  to  make  the 
display  clearer  by  dimming  the  tablet 
illumination  lights  under  the  mirror. 

It  is  startling  to  realize  how  minimally 
visible  the  hand  must  be  for  use  of  the 
tablet — people  rapidly  relied  upon  the 
display  image  to  obtain  feedback  of 
their  hand  and  pen  positions. 

In  terms  of  implementation,  there  are 
several  areas  for  improvement.  The 
weakest  and  most  critical  link  is  the 
television  projector  and  rear  projection 
screen.  Because  of  the  high  cost  of 
projection,  it  is  the  major  impediment  to 
adoption  of  the  device.  Unfortunately, 
it  is  currently  a  necessary  component 
of  our  interactive  device,  both  to  provide 


the  requisite  brightness  and  flatness. 

There  is,  however,  some  hope  for  substitutes 
in  the  near  future.  Broader  user  interest 
in  larger  displays  has  generated  industry 
investigations  into  less  expensive 
alternatives.  Since  the  image  for  our 
device  is  not  extremely  large  (only  20" 
square),  direct  projecting  color  CRT's 
are  possible,  and  directly  viewable 
flat  panels  are  closer  to  realization. 

Both  new  approaches  to  flat  display 
design  have  the  advantage  that  they  would 
permit  a  more  compact,  more  easily 
adjustable  implementation  of  our  ideas. 

With  the  flat  panel,  the  device  could  be 
constructed  as  a  small  portable  structure 
for  use  on  a  desktop. 

The  touch  panel  part  of  the  design  has 

disadvantages  and  advantages.  Common 

touch  panels  are  directed  toward 

applications  where  simple  pointing 
2 

suffices.  They  normally  do  not  allow 
hands  to  be  rested  on  them  (because  of 
the  resulting  touch  signal) ,  and  resolution 
is  not  a  highly  prized  goal. 

However,  Elographics  has 
recently  designed  a  device  that  detects 
touch  only  with  high  forces  per  unit 
area.  Thus  a  hand  may  rest  on  the 
surface  and  be  invisible  to  the  computer, 
but  the  high  load  imposed  by  a  penpoint 
will  be  detected.  This  will  allow  the 


The  unavailability  of 
a  tracking  cursor  on 
touch  panels  need  not 
be  a  disadvantage  to 
the  user.  Just  as  the 
near-field  was 
disregarded  in  the 
digitizing  tablet,  it 
is  not  necessary 
with  a  touch  panel.  In 
short,  a  cursor  is  not 
necessary  for  accurate 
display  interaction. 
Cursors  are  an  artifact 
of  the  disjoint  input 
and  display  surfaces  in 
other  systems  and 
provide  feedback  as  to 
exact  pen  position.  In 
this  system,  that  feedback 
is  automatic,  and  need 
not  be  provided 
artificially. 


use  of  normal,  inactive  stylii,  on 
a  high  resolution  surface.  The 
combination  of  inexpensive,  flat  displays 
and  Elographics  type  tablet  would 
result  in  a  highly  flexible  and  easy 
to  use  interactive  graphic  device. 
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